Methods and systems for multi-pattern searching

ABSTRACT

Embodiments of the present invention relate to systems and methods for optimizing and reducing the memory requirements of state machine algorithms in pattern matching applications. Memory requirements of an Aho-Corasick algorithm are reduced in an intrusion detection system by representing the state table as three separate data structures. Memory requirements of an Aho-Corasick algorithm are also reduced by applying a banded-row sparse matrix technique to the state transition table of the state table. The pattern matching performance of the intrusion detection system is improved by performing a case insensitive search, where the characters of the test sequence are converted to uppercase as the characters are read. Testing reveals that state transition tables with sixteen bit elements outperform state transition tables with thirty-two bit elements and do not reduce the functionality of intrusion detection system using the Aho-Corasick algorithm.

BACKGROUND OF THE INVENTION

1. Field of the Invention

Embodiments of the present invention relate to methods and systems forefficiently searching for patterns in a character stream. Moreparticularly, embodiments of the present invention relate to systems andmethods for optimizing and reducing the memory requirements of statemachine algorithms in pattern matching applications.

2. Background Information

An intrusion detection system (IDS) examines network traffic packets. AnIDS searches for intruders by looking for specific values in the headersof packets and by performing a search for known patterns in theapplication data layer of packets. Typical IDSs rely heavily on theAho-Corasick multi-pattern search engine. As a result, the performancecharacteristics of the Aho-Corasick algorithm have a significant impacton the overall performance of these IDSs.

The pattern search problem in IDSs is a specialized problem thatrequires consideration of many issues. The real-time nature ofinspecting network packets requires the use of a pattern search enginethat can keep up with the speeds of modern networks. There are two typesof data used in this type of search. These two types of data are thepatterns and the search text. The patterns are pre-defined and static.This allows them to be pre-processed into the best form suitable for anygiven pattern matching algorithm. The search text is dynamicallychanging as each network packet is received. This prohibits thepre-processing of the search text prior to performing the patternsearch. This type of pattern search problem is defined as a serialpattern search. The Aho-Corasick algorithm is a classic serial searchalgorithm, and was first introduced in 1975. Many variations have beeninspired by the Aho-Corasick algorithm, and more recently IDSs havegenerated a renewed interest in the algorithm due to some of thealgorithm's unique properties.

IDSs search for patterns that can be either case sensitive or caseinsensitive. The Aho-Corasick algorithm was originally used as a casesensitive pattern search engine. The size of the patterns used in asearch can significantly affect the performance characteristics of thesearch algorithm. State machine based algorithms, such as theAho-Corasick algorithm, are not affected by the size of the smallest orlargest pattern in a group. Skip based methods, such as the Wu-Manberalgorithm and others that utilize character skipping features, are verysensitive to the size of the smallest pattern. The ability to skipportions of a search text can greatly accelerate the search enginesperformance. However, skip distance is limited by the smallest pattern.Search patterns in IDSs represent portions of known attack patterns andcan vary in size from one to thirty or more characters but are usuallysmall.

The pattern group size usually affects IDS performance because the IDSpattern search problem typically benefits from processor memory caching.Small pattern groups can fit within the cache and benefit most from ahigh performance cache. As pattern groups grow larger, less of thepattern group fits in the cache, and there are more cache misses. Thisreduces performance. Most search algorithms will perform faster with tenpatterns than they do with one-thousand patterns, for instance. Theperformance degradation of each algorithm as pattern group sizeincreases varies from algorithm to algorithm. It is desirable that thisdegradation be sub-linear in order to maintain scalability.

The alphabet size used in current IDSs is defined by the size of a byte.An eight bit byte value is in the range 0 to 255, providing IntrusionDetection Systems with a 256-character alphabet. These byte valuesrepresent the ASCII and control characters seen on standard computerkeyboards and other non-printable values. For instance, the letter ‘A’has a byte value of sixty-five. Extremely large alphabets such asUnicode can be represented using pairs of byte values, so the alphabetthe pattern search engine deals with is still 256 characters. This is alarge alphabet by pattern matching standards. The English dictionaryalphabet is fifty-two characters for upper and lower case characters,and DNA research uses a four character alphabet in gene sequencing. Thesize of the alphabet has a significant impact on which search algorithmsare the most efficient and the quickest.

Algorithmic attacks by intruders attempt to use the properties andbehaviors of a search algorithm against itself to reduce the searchalgorithm's performance. The performance behavior of a search algorithmshould be evaluated by considering the search algorithm's average-caseand worst-case performance. Search algorithms that exhibit significantlydifferent worst-case and average-case performance are susceptible toalgorithmic attacks by intruders. Skip based search algorithms, such asthe Wu-Manber algorithm, utilize a character skipping feature similar tothe bad character shift in the Boyer-Moore algorithm. Skip based searchalgorithms are sensitive to the size of the smallest pattern since theycan be shown to be limited to skip sizes smaller than the smallestpattern. A pattern group with a single byte pattern cannot skip overeven one character or it might not find the single byte pattern.

The performance characteristics of the Wu-Manber algorithm can besignificantly reduced by malicious network traffic, resulting in aDenial of Service attack. This type of attack requires degeneratetraffic of small repeated patterns. This problem does not exist for textsearches where algorithmic attacks are not intentional and are usuallyvery rare. It is a significant issue in IDSs where algorithmic attacksare prevalent and intentional. The Wu-Manber algorithm also does notachieve its best performance levels in a typical IDS, because IDS rulesoften include very small search patterns of one to three characters,eliminating most of the opportunities to skip sequences of characters.

In practice, an IDS using the Wu-Manber algorithm provides littleperformance improvement over an IDS using the Aho-Corasick algorithm inthe average-case. However, an IDS using the Wu-Manber algorithm can besignificantly slower than an IDS using the Aho-Corasick algorithm whenexperiencing an algorithmically designed attack. The strength ofskip-based algorithms is evident when all of the patterns in a group arelarge. The skip-based algorithms can skip many characters at a time andare among the fastest average-case search algorithms available in thiscase. The Aho-Corasick algorithm is unaffected by small patterns. TheAho-Corasick algorithm's worst-case and average-case performance are thesame. This makes it a very robust algorithm for IDSs.

The size of search text read by in IDSs is usually less than a fewthousand bytes. In general when searching text, the expense of settingup the search pattern and filling the cache with the necessary searchpattern information is usually fixed. If the search is performed overjust a few bytes, then spreading the setup costs over those few bytesresults in a high cost per byte. Whereas, if the search text is verylarge, spreading the setup cost over the larger search text results invery little overhead cost added to searching each byte of text.

The frequency of searching in an Intrusion Detection System is dependenton the network bandwidth, the volume of traffic on the network, and thesize of the network packets. This implies that the frequency of patternsearches and the size of each search text are related due to the natureof the network traffic being searched. Again, as with search text size,a high frequency of searching in an IDS will cause the search setupcosts to be significant compared to doing fewer larger text searches.

In view of the foregoing, it can be appreciated that a substantial needexists for methods and systems that take into consideration the manyissues associated with pattern searching in IDSs and improve the overallperformance of IDSs.

BRIEF SUMMARY OF THE INVENTION

One embodiment of the present invention is a method for building a statetable of a state machine algorithm in a pattern matching application. Atleast one search pattern is identified. A search pattern trie iscreated. The search pattern trie is made up of a list of all validcharacters for each state. Each element of the list of all validcharacters includes a valid character and a next valid state pair. Atleast one search pattern is added to the search pattern trie. The searchpattern is added one character at a time to the search pattern trie,where each character represents a state. A non-deterministic finiteautomata is built from the search pattern trie. The search pattern trieis replaced with the non-deterministic finite automata in a list format.A deterministic finite automata is built from the non-deterministicfinite automata. The non-deterministic finite automata is replaced withthe deterministic finite automata in a list format. The deterministicfinite automata is converted into three separate data structures. Thesethree data structures include a state transition table, an array of perstate matching pattern lists, and a separate failure pointer list foreach state for the non-deterministic finite automata. These three datastructures compose the state table.

Another embodiment of this method involves converting a portion of thedeterministic finite automata into a state transition table. A statevector for each state is allocated. A state transition element of thestate vector is created for each element of the list of all validcharacters. The next valid state of each element of the list of allvalid characters is copied into each of the state transition elements ofthe state vector. In this embodiment, the state transition elements arestored in full vector form.

In another embodiment of this method, the state transition elements arestored in banded-row form. A band is defined as the group of elementsfrom the first nonzero state from the list of all valid characters tothe last nonzero state from the list of all valid characters. A statevector is allocated for each state. A state transition element of thestate vector is created for each element of the band. An element of thestate vector is created to identify an index of the first nonzero statefrom the list of all valid characters. An element of the state vector iscreated to identify the number of elements in the band. The state ofeach element of the band is copied into each state transition element ofthe state vector.

In another embodiment of this method, an element is added to the statevector to identify the format of the state vector. Exemplary formatsinclude but are not limited to full vector and banded vector. In anotherembodiment of this method, an element is added to the state vector toidentify if the state vector includes a matching pattern state.

Another embodiment of the present invention is a method for searchingfor search patterns in a text sequence using a state table of a statemachine algorithm in a pattern matching application. A character fromthe text sequence is read. An element of a current state vector of thestate table that corresponds to the character is read. If the element isnonzero, a pattern matching element of the current state vector for amatching pattern flag is checked. If the matching pattern flag is set, apattern match is logged. The current state vector corresponding to thevalue of the element is selected. The next character from the textsequence is then read.

In another embodiment of this method, an element of the current statevector that identifies a format is checked. If the format is banded, itis determined if the character is in the band. This determination ismade using the element of the current state vector identifying a numberof elements in the band, the element of the current state vectoridentifying an index of a first element of the band, and elements of theband. If the character is in the band, the current state vectorcorresponding to the value of the element is selected. If the characteris not in the band, the current state vector is set to the initialstate.

Another embodiment of the present invention is a state table for a statemachine algorithm used in a pattern matching application. The statetable includes a state transition table, an array of per state matchingpattern lists, and a failure pointer list for each state for anon-deterministic finite automata. The state transition table includes astate vector for each state. In one embodiment of this state table, thestate vector is stored in a sparse vector format. Sparse vector formatsinclude but are not limited to the compressed sparse vector format, thesparse-row format, and the banded-row format. The state machinealgorithm is the Aho-Corasik algorithm, for example. The state table andstate machine algorithm are used in pattern matching applicationsincluding but not limited to intrusion detection systems, passivenetwork monitors, active network monitors, protocol analyzers, andsearch engines.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a flowchart showing a method for building a state table of astate machine algorithm in a pattern matching application, in accordancewith an embodiment of the present invention.

FIG. 2 is a flowchart showing a method for searching for search patternsin a text sequence using a state table of a state machine algorithm in apattern matching application, in accordance with an embodiment of thepresent invention.

FIG. 3 is schematic diagram of the components of a state table for astate machine algorithm used in a pattern matching application, inaccordance with an embodiment of the present invention.

FIG. 4 is a bar graph showing exemplary test results for a dictionarysearch using the standard Aho-Corasick algorithm used in Snort, anembodiment of the present invention using full matrix storage, and anembodiment of the present invention using banded-row storage allcompiled with an Intel™ C 6.0 compiler.

FIG. 5 is a bar graph showing exemplary test results for a dictionarysearch using the standard Aho-Corasick algorithm used in Snort, anembodiment of the present invention using full matrix storage, and anembodiment of the present invention using banded-row storage allcompiled with a Cygwin gcc 3.3.1 compiler.

FIG. 6 is a bar graph showing exemplary test results for a dictionarysearch using the standard Aho-Corasick algorithm used in Snort, anembodiment of the present invention using full matrix storage, and anembodiment of the present invention using banded-row storage allcompiled with a Microsoft™ Visual C++™ 6.0 compiler.

FIG. 7 is a bar graph showing exemplary test results for a dictionarysearch using the standard Aho-Corasick algorithm used in Snort, anembodiment of the present invention using full matrix storage, and anembodiment of the present invention using banded-row storage allcompiled with an Intel™ C 6.0 compiler and normalized to the results ofthe standard Aho-Corasick algorithm used in Snort.

FIG. 8 is a bar graph showing exemplary test results for a dictionarysearch using the standard Aho-Corasick algorithm used in Snort, anembodiment of the present invention using full matrix storage, and anembodiment of the present invention using banded-row storage allcompiled with a Cygwin gcc 3.3.1 compiler and normalized to the resultsof the standard Aho-Corasick algorithm used in Snort.

FIG. 9 is a bar graph showing exemplary test results for a dictionarysearch using the standard Aho-Corasick algorithm used in Snort, anembodiment of the present invention using full matrix storage, and anembodiment of the present invention using banded-row storage allcompiled with a Microsoft™ Visual C++™ 6.0 compiler and normalized tothe results of the standard Aho-Corasick algorithm used in Snort.

FIG. 10 is a bar graph showing exemplary storage requirements for adictionary search using the standard Aho-Corasick algorithm used inSnort, an embodiment of the present invention using full matrix storage,and an embodiment of the present invention using banded-row storage.

FIG. 11 is a table showing exemplary storage requirements for adictionary search the standard Aho-Corasick algorithm used in Snort, anembodiment of the present invention using full matrix storage, and anembodiment of the present invention using banded-row storage.

FIG. 12 is a bar graph showing exemplary test results for a dictionarysearch using an embodiment of the present invention using full matrixstorage with sixteen bit and thirty-two bit state values both compiledwith an Intel™ C 6.0 compiler.

FIG. 13 is a bar graph showing exemplary test results for a dictionarysearch using an embodiment of the present invention using full matrixstorage with sixteen bit and thirty-two bit state values both compiledwith a Cygwin gcc 3.3.1 compiler.

FIG. 14 is a bar graph showing exemplary test results for a dictionarysearch using an embodiment of the present invention using full matrixstorage with sixteen bit and thirty-two bit state values both compiledwith a Microsoft™ Visual C++™ 6.0 compiler.

FIG. 15 is a bar graph showing exemplary test results for a dictionarysearch using an embodiment of the present invention using banded-rowstorage with sixteen bit and thirty-two bit state values both compiledwith an Intel™ C 6.0 compiler.

FIG. 16 is a bar graph showing exemplary test results for a dictionarysearch using an embodiment of the present invention using banded-rowstorage with sixteen bit and thirty-two bit state values both compiledwith a Cygwin gcc 3.3.1 compiler.

FIG. 17 is a bar graph showing exemplary test results for a dictionarysearch using an embodiment of the present invention using banded-rowstorage with sixteen bit and thirty-two bit state values both compiledwith a Microsoft™ Visual C++™ 6.0 compiler.

FIG. 18 is a table showing exemplary test results a search of a twoG-byte file of Web-centric network traffic using a Wu-Manber algorithm,the standard Aho-Corasick algorithm used in Snort, an embodiment of thepresent invention using full matrix storage, and an embodiment of thepresent invention using banded-row storage.

FIG. 19 is a bar graph showing exemplary test results a search of a twoG-byte file of Web-centric network traffic using a Wu-Manber algorithm,the standard Aho-Corasick algorithm used in Snort, an embodiment of thepresent invention using full matrix storage, and an embodiment of thepresent invention using banded-row storage.

Before one or more embodiments of the invention are described in detail,one skilled in the art will appreciate that the invention is not limitedin its application to the details of construction, the arrangements ofcomponents, and the arrangement of steps set forth in the followingdetailed description or illustrated in the drawings. The invention iscapable of other embodiments and of being practiced or being carried outin various ways. Also, it is to be understood that the phraseology andterminology used herein is for the purpose of description and should notbe regarded as limiting.

DETAILED DESCRIPTION OF THE INVENTION The Aho-Corasick State Machine

The Aho-Corasick state machine is a specialized finite state machine. Afinite state machine is a representation of all of the possible statesof a system, along with information about the acceptable statetransitions of the system. The processing action of a state machine isto start in an initial state, accept an input event, and move thecurrent state to the next correct state based on the input event. It ispossible to model a state machine as a matrix where the rows representstates and the columns represent events. The matrix elements provide thenext correct state to move to based on the input event and possibly somespecific action to be done or information to be processed when the stateis entered or exited. For example, if the current state is state ten andthe next input event is event six, then to perform a state transition,the matrix element at row ten and column six is examined. The currentstate ten is changed to the state indicated by the value of the matrixelement at row ten and column six.

An Aho-Corasick state machine is implemented in a typical IDS as adeterministic finite automata (DFA). A unique property of a DFA is that,upon examining an input character in the text stream, it requiresexactly one transition of the state machine to find the correct nextstate. This is in contrast to a non-deterministic finite automata (NFA),which can require more than one state transition to find the correctnext state. A DFA can process data faster than an NFA since it requiresfewer transition steps. An NFA can require up to twice as manytransitions as a DFA to search a stream of data. The construction andstructure of the state table matrix of a DFA is also more complicatedthan that of an NFA. A DFA can be constructed from an NFA bypre-processing all of the possible state transitions in the NFA until itcan be determined how to perform all state transitions in a single step.This procedure has the tendency to fill in more elements of the statetable matrix. The physical representation of the Aho-Corasick statetransition table varies from one implementation to the next. The choiceof the state table representation determines the memory and performancetradeoffs of the search algorithm.

Sparse Storage Formats

Sparse matrices and vectors contain a significant number of zeroelements and only a few non-zero elements. Methods and storage formatsused for operating on sparse structures efficiently are well developedin the field of linear algebra and utilized in many branches of science.It is not sufficient to store sparse data efficiently. The storageformat must also allow for fast random access to the data.

A sample sparse matrix is shown below. It contains six rows and fourcolumns.

$\begin{matrix}{0\mspace{11mu} 0\mspace{11mu} 0\mspace{11mu} 3} \\{0\mspace{11mu} 4\mspace{11mu} 0\mspace{11mu} 1} \\{0\mspace{11mu} 0\mspace{11mu} 0\mspace{11mu} 6} \\{1\mspace{11mu} 0\mspace{11mu} 0\mspace{11mu} 0} \\{{0\mspace{11mu} 2\mspace{11mu} 0\mspace{11mu} 0}\;} \\{0\mspace{11mu} 0\mspace{11mu} 5\mspace{11mu} 0}\end{matrix}$

The above matrix is a general rectangular matrix with only a fewnon-zero elements. Using the compressed row storage (CRS) format it canbe stored as shown below.

$\begin{matrix}{{Value}\text{:}} & {3\mspace{11mu} 4\mspace{11mu} 1\mspace{11mu} 6\mspace{11mu} 1\mspace{11mu} 2\mspace{11mu} 5} \\{{Column}\text{:}} & {4\mspace{11mu} 2\mspace{11mu} 4\mspace{11mu} 4\mspace{11mu} 1\mspace{11mu} 2\mspace{11mu} 3} \\{{Row}\text{:}} & {{1\mspace{11mu} 2\mspace{11mu} 4\mspace{11mu} 5\mspace{11mu} 6\mspace{11mu} 7}\mspace{20mu}}\end{matrix}$

This CRS format requires three vectors to hold only the non-zeroentries. They are a value vector, a column vector and a row vector. Thecolumn vector indicates the column that the corresponding value in thevalue array belongs to in the matrix. The row vector indicates the indexin the Column and Value arrays where each row starts. There are sevenentries in each of the value and column vectors, since there are sevennon-zero entries in the matrix. There are six row entries, since thereare six rows in the matrix. For example, the third row entry indicatesthe third row starts at the fourth entry in the column and valuevectors. This is one of the simplest sparse storage schemes for amatrix, which can be broken down further to handle vectors byconsidering a single row to be a vector as shown below.

The format shown below is called the compressed sparse vector format.Each vector in this format corresponds to a row of a matrix.

$\begin{matrix}{{Vector}\text{:}} & {{0\mspace{11mu} 0\mspace{11mu} 0\mspace{11mu} 2\mspace{11mu} 4\mspace{11mu} 0\mspace{11mu} 0\mspace{11mu} 0\mspace{11mu} 6\mspace{11mu} 0\mspace{11mu} 7\mspace{11mu} 0\mspace{11mu} 0\mspace{11mu} 0\mspace{11mu} 0\mspace{11mu} 0\mspace{11mu} 0\mspace{11mu} 0\mspace{11mu} 0\mspace{11mu} 0}\mspace{11mu}} \\{{Value}\text{:}} & {{2\mspace{11mu} 4\mspace{11mu} 6\mspace{11mu} 7}\mspace{304mu}} \\{{Index}\text{:}} & {{4\mspace{11mu} 5\mspace{11mu} 9\mspace{11mu} 11}\mspace{295mu}}\end{matrix}$

In this compressed sparse vector format, the index array indicates thevector array index of the corresponding value. There are four non-zerovalues, hence there are four entries in the index and value arrays.

Merging the value and index arrays together into one integer arrayresults in the format shown below. This format is called the sparse-rowformat.

-   -   Sparse-Row: 8 4 2 5 4 9 6 11 7

In this sparse-row format there are nine entries, the first is the totalnumber of words that follows, followed by eight numbers representingfour pairs of numbers in index-value order. This is a single array ofnine entries, eight in index-value order. This could also be representedas an array of four C language structures each having an Index and aValue entry. In either case, only the number of non-zero entries and onearray of the non-zero entries is needed. This storage scheme works wellat minimizing the total storage of a sparse vector and works well inapplications where every element of the vector must be touchedsequentially, such as in matrix-vector multiplication. However, forrandomly looking up an individual entry, this format requires a searchthrough the array to find the correct index.

Another representation of a vector uses the banded nature of the vectorto store the elements efficiently. This representation also allowsrandom access to the vector elements to be maintained. Thisrepresentation is called the banded-row format and is shown below.

$\begin{matrix}{{Num}\mspace{14mu} {Terms}\text{:}} & 8 \\{{Start}\mspace{14mu} {Index}\text{:}} & 4 \\{{Values}\text{:}} & {2\mspace{11mu} 4\mspace{11mu} 0\mspace{11mu} 0\mspace{11mu} 0\mspace{11mu} 6\mspace{11mu} 0\mspace{11mu} 7} \\{{Band}\mspace{14mu} {Array}\text{:}} & {8\mspace{11mu} 4\mspace{11mu} 2\mspace{11mu} 4\mspace{11mu} 0\mspace{11mu} 0\mspace{11mu} 0\mspace{11mu} 6\mspace{11mu} 0\mspace{11mu} 7}\end{matrix}$

This banded-row format stores elements from the first non-zero value tothe last non-zero value. The number of terms stored is known as thebandwidth of the vector. Small bandwidth corresponds to large storagesavings. To manage access to the data, only the number of data elementsand the starting index of the data are tracked. This format reduces thestorage requirements, and still provides fast random access to the data.Many problems express themselves as banded matrices, in which case thebanded-row format can be used for each row in the matrix. This type ofbanded storage puts no requirements on how the banded-ness of one rowwill correspond to banded-ness in another row.

The Optimized Aho-Corasick State Machine

One embodiment of the present invention is an optimized Aho-Corasickstate machine designed for use in an IDS. In this embodiment, the statetable is managed somewhat differently, which allows the search routineto be compiled to a more optimal instruction mix. In a typical IDS usingthe un-optimized Aho-Corasick state machine algorithm, each entry in thestate table has a vector of transitions for the state, a failure pointerfor the NFA version of the table, and a list of matching patterns forthe state, all contained in one structure. In this embodiment, the statetable is broken into a state transition table, an array of per statematching pattern lists, and a separate failure pointer list for eachstate for the NFA. In other words, the state table is broken into threeseparate data structures. This division of the state table into threeseparate data structures improves performance. The size of the statetransition table is reduced, which allows the state transition table tobe manipulated in memory cache rather than main memory, and thereforeimproves the overall performance of the state machine algorithm.

The state transition table is a list of pointers to state vectors. Eachstate vector represents the valid state transitions for that state. Inanother embodiment of the present invention, each vector also includes asmall header that indicates the storage format of the row, and a Booleanflag indicating if any pattern matches occur in this state. The statetransition data follows the header. This embodiment also improvesperformance. Since there is a Boolean flag indicating if any patternmatches occur, a list of matching patterns no longer needs to betraversed at each state transition.

In another embodiment of the present invention, the elements of thestate transition table are reduced in size from four bytes to two bytes.Experimentation has revealed that the number of states used by a typicalIDS does not exceed sixty-five thousand states, or the maximum number ofstates that can be represented by two bytes. Previously, four bytes wereused, allowing four billion possible states. Again, the use of two bytevector elements reduces memory consumption and improves overallperformance.

In another embodiment of the present invention, each input character isconverted to uppercase as it is processed in the search, rather thanconverting the entire input search text prior to the search. Convertingthe entire input search text requires that uppercase search text bestored in main memory, or, at least, in cache memory. Thus, byconverting each input character as it is processed, no memory isrequired for the uppercase search text in this embodiment. Caseconversion takes place in a computer register, for example. As a result,the processing time is reduced.

These above embodiments were shown in tests to significantly improveperformance over the un-optimized Aho-Corasick state machine used in atypical IDS.

A search routine is a small piece of code that essentially jumps fromstate to state based on the input characters in the text stream. A statemachine operates by maintaining a current state, accepting an inputevent (a character in our case), and using a table or matrix to lookupthe correct next allowed state based on the current state and inputevent as shown below in psuedo-computer code.

while ( input = next-input ) { state = state-transition-table[ state,input ] if( patterns matched in this state ) process patterns.... }

The C language code below shows the basic algorithm for the search usedby the optimized Aho-Corasick algorithm, in accordance with anembodiment of the present invention.

for ( state = 0; T < Tend; T++ ) { ps = NextState[ state ]; sindex =xlatcase[ T[0] ]; if( ps[1] ) { for( mlist = MatchList[state]; mlist !=NULL; mlist = mlist−>next ) { /* process the pattern match */ } } state= ps[ 2u + sindex ]; }

The NextState array is the array of pointers to the row vectors of thestate table. The ‘ps’ variable is used to point to the current state'soptimized storage vector. The optimized storage vector includes thestorage format indicator, the Boolean pattern match flag, and statetransition vector information. The first two words of the ‘ps’ array arethe storage format and the Boolean match flag. The ‘T’ parameter is thetext being searched, and ‘xlatcase’ converts the text one byte at a timeto upper case for a case independent search. Once a match is found, thepattern is processed. A check is made using the ‘ps’ variable todetermine if any patterns are matched in the current state. If nopatterns are matched, the state machine cycles to the next state. Ifthere is a match, all of the patterns that match in the current stateare processed in the ‘for’ loop.

The Sparse Storage Optimized Aho-Corasick State Machine

In another embodiment of the present invention, a banded-row sparsestorage format is used in the optimized Aho-Corasick state machine. Thevectors of the state transition table are as follows for the banded-rowformat. The first word indicates the storage format. The second wordindicates if any patterns match in this state. The third word indicatesthe number of terms that are stored for the row. Finally, the fourthword indicates the index of the first term. The banded-row format allowsdirect index access to an entry. However, a bounds check must be doneprior to each indexing operation. A C language example of this algorithmis shown below.

for ( state = 0; T < Tend; T++ ) { ps = NextState[ state ];  sindex =xlatcase[ T[0] ];  if( ps[1] ) { for( mlist = MatchList[ state ]; mlist!= NULL; mlist = mlist−>next ) { /* process the pattern match */ } } /*Bounds check & state transition */ if( sindex < ps[3] ) state = 0; elseif( sindex >= (ps[3] + ps[2]) ) state = 0; else state = ps[ 4u + sindex− ps[3] ]; }

The NextState array is the array of pointers to the row vectors of thestate table. The ‘ps’ variable is used to point to the current state'soptimized storage vector. The optimized storage vector includes thestorage format indicator, the Boolean pattern match flag, and the statetransition vector information. The first two words of the ‘ps’ array arethe storage format and the Boolean match flag. The ‘T’ parameter is thetext being searched, and ‘xlatcase’ converts the text one byte at a timeto upper case for a case independent search. The uppercase character isthe input-index used in indexing into the banded storage vector. A checkis made using the second word of the ‘ps’ variable to determine if anypatterns are matched in the current state (indexing starts with thefirst word having index of 0). If no patterns are matched, the statemachine cycles to the next state. The next state is determined by firstperforming a bounds check, and then the indexing formula‘index=4+input-index-start-index’ is applied to calculate the correctoffset of the desired ‘input-index.’ If there is a match, all of thepatterns that match in the current state are processed in the ‘for’loop.

Building a State Table

FIG. 1 is a flowchart showing a method 100 for building a state table ofa state machine algorithm in a pattern matching application, inaccordance with an embodiment of the present invention. The statemachine algorithm is the Aho-Corasick algorithm, for example. Patternmatching applications include but are not limited to intrusion detectionsystems, passive network monitors, active network monitors, protocolanalyzers, and search engines.

In step 110 of method 100, at least one search pattern is identified. Inanother embodiment of this method, the characters of the search patternare converted to uppercase so that the initial search is caseinsensitive.

In step 120, a search pattern trie is created. The search pattern trieis made up of a list of all valid characters for each state. Eachelement of the list of all valid characters includes a valid characterand a next valid state pair. In another embodiment of this method, thelist of all valid characters for each state includes all uppercasecharacters for each state. The allows a case insensitive pattern search.

In step 130, at least one search pattern is added to the search patterntrie. The search pattern is added one character at a time to the searchpattern trie, where each character represents a state.

In step 140, an NFA is built from the search pattern trie. The searchpattern trie is replaced with the NFA in a list format. The NFA is builtfrom a search pattern trie and placed in list format according towell-known methods.

In step 150, a DFA is built from the NFA. The NFA is replaced with theDFA in a list format. The DFA is built from the NFA and placed in listformat according to well-known methods.

In step 160, the DFA is converted into three separate data structures.These three data structures include a state transition table, an arrayof per state matching pattern lists, and a separate failure pointer listfor each state for the NFA. Previously, each entry in the state tablehad a vector of transitions for the state, a failure pointer for the NFAversion of the table, and a list of matching patterns for the state, allcontained in one structure.

In step 170, the three data structures are used as the state table.

The entries of the DFA are converted to a state transition table. Astate vector is allocated for each state. In another embodiment of thismethod, each element of the state vector is sixteen bits. There is astate transition element of the state vector for each element of thelist of all valid characters. The state of each element of the list ofall valid characters is copied into each of the elements of the statevector. In another embodiment of this method, there are two-hundred andfifty-six state transition elements in the state vector.

In another embodiment of this method, an element of the state vectoridentifies the format of the state vector. Formats include but are notlimited to full and banded. In another embodiment of this method, anelement of the state vector identifies if the state vector contains amatching pattern state.

In another embodiment of this method, the state transition elements arestored in banded-row form. A band is defined as the group of elementsfrom the first nonzero state from the list of all valid characters tothe last nonzero state from the list of all valid characters. A statevector is allocated for each state. In another embodiment of thismethod, each element of the state vector is sixteen bits. A statetransition element of the state vector is created for each element ofthe band. An element of the state vector is created to identify an indexof the first nonzero state from the list of all valid characters. Anelement of the state vector is created to identify the number ofelements in the band. The state of each element of the band is copiedinto each state transition element of the state vector.

In another embodiment of this method, an element of the state vectoridentifies the format of the state vector. In another embodiment of thismethod, an element of the state vector identifies if the state vectorcontains a matching pattern state.

In one embodiment of the present invention, the full vector format of astate vector consists of two-hundred and fifty-eight elements. Eachelement is sixteen bits. The first element identifies the storageformat. These formats include but are not limited to full and banded.The second element identifies if any patterns match in this statevector. The remaining two-hundred and fifty-six elements are thetransition elements.

In another embodiment of the present invention, the sparse banded vectorformat of a state vector consists of a variable number of elements, butnot more than three-hundred. Each element is sixteen bits. The firstelement identifies the storage format. These formats include but are notlimited to full and banded. The second element identifies if anypatterns match in this state vector. The third element is the number ofelements in a band. The fourth element is the index of the first elementin the band. The remaining elements are the transition elements.

In another embodiment of the present invention, the state table is builtfirst by adding each pattern to a list of patterns. Each pattern isconverted to upper case and stored for future use in the search engine.This is an initial step to produce a case insensitive search, instead ofa typical case sensitive search produced by the typical Aho-Corasickalgorithm. The original and uppercase patterns are kept in memory andboth are available during the search phase. Each pattern from the listof patterns is added to the state table. Each pattern is added onecharacter at a time, in uppercase, to the state table. Characters areadded to a list for each state that stores all of the valid uppercasecharacters for the state. The list also contains a next valid stateparameter for each character. The next valid state is used to direct thesearch to the next state. This list of the state table is a searchpattern trie using a list format.

An NFA is built from the trie, replacing the trie list with the NFA inlist format. The construction of the NFA follows the standardAho-Corasick construction for an NFA. A DFA is built from the NFA,replacing the NFA with the DFA in list format. The construction of theDFA follows the standard Aho-Corasick construction for a DFA.

The DFA is then converted to a full vector format or a banded vectorformat. For the full vector format, a state vector with two-hundred andfifty-eight elements is created for each state. Each state vector holdsa format element, a Boolean pattern match element, and two-hundred andfifty six state transition elements. For each state vector, the formatof the state vector is made full by setting the format field value tozero. For each transition element of the state vector, the Booleanpattern match flag is set as matching a pattern or not matching apattern, using one or zero, respectively. The two-hundred and fifty-sixtransition elements are initialized to zero for each state. The list ofvalid state transitions from the DFA are copied to their respectivepositions in the state element data area. For example, the letter ‘A’has a value of sixty-five, and the state for ‘A’ would be placed in thesixty-fifth position of the transition elements.

For the banded vector format, a state vector with a variable number ofelements is created for each state. Each state vector holds a formatelement, a Boolean pattern match element, a number of elements in theband element, an index of the first element in the band, and the band ofstate transition elements. The list of valid state transitions from theDFA is examined. The elements from the first nonzero element to the lastnonzero element are defined as the band. The band is copied to the statetransition elements of the state vector. The number of elements in theband element and index of the first element in the band are also storedin the state vector. The format of the state vector is set to banded orone. For each transition element of the state vector, the Booleanpattern match flag is set as matching a pattern or not matching apattern, using one or zero, respectively.

Searching for Search Patterns in a Text Sequence

FIG. 2 is a flowchart showing a method 200 for searching for searchpatterns in a text sequence using a state table of a state machinealgorithm in a pattern matching application, in accordance with anembodiment of the present invention. The state machine algorithm is theAho-Corasick algorithm, for example. Pattern matching applicationsinclude but are not limited to intrusion detection systems, passivenetwork monitors, active network monitors, protocol analyzers, andsearch engines.

In step 210 of method 200, a character from the text sequence is read.In one embodiment of this method, the character is converted to uppercase. This conversion allows a case insensitive search.

In step 220, an element of a current state vector of the state tablethat corresponds to the character is read.

In step 230, if the element is nonzero, a pattern matching element ofthe current state vector for a matching pattern flag is checked.

In step 240, if the matching pattern flag is set, a pattern match islogged.

In step 250, the current state vector corresponding to the value of theelement is selected.

In step 260, the next character from the text sequence is then read.

In another embodiment of this method, an element of the current statevector that identifies a format is checked. If the format is banded, itis determined if the character is in the band. This determination ismade using the element of the current state vector identifying a numberof elements in the band, the element of the current state vectoridentifying an index of a first element of the band, and elements of theband. If the character is in the band, the current state vectorcorresponding to the value of the element is selected. If the characteris not in the band, the current state vector is set to the initialstate.

In another embodiment of this method, if the matching pattern flag isset, a case sensitive version of the pattern match from the textsequence and a case sensitive version of a search pattern are compared.If the case sensitive version of the pattern match from the textsequence and the case sensitive version of a search pattern match, thecase sensitive version of the pattern match is logged.

State Table

FIG. 3 is schematic diagram of the components of a state table 300 for astate machine algorithm used in a pattern matching application, inaccordance with an embodiment of the present invention. State table 300includes state transition table 320, an array of per state matchingpattern lists 330, and failure pointer list for each state for a NFA340. State transition table 320 includes a state vector for each state.In one embodiment of state table 300, the state vector is stored in asparse vector format. Sparse vector formats include but are not limitedto the compressed sparse vector format, the sparse-row format, and thebanded-row format. The state machine algorithm is the Aho-Corasikalgorithm, for example. State table 300 and the state machine algorithmare used in pattern matching applications including but not limited tointrusion detection systems, passive network monitors, active networkmonitors, protocol analyzers, and search engines.

Performance Metrics

Algorithmic or theoretical metrics are based on consideration of thealgorithm independent of the hardware or software. Worst-case behavioris an example of an algorithmic metric. Typically, the worst-caseperformance of a multi-pattern search engine is proportional to the sizeof the patterns and the length of the data being searched. Oneembodiment of the present invention is an O(n) algorithm for an IDS,indicating the search speed is proportional to n, the length of the databeing searched.

Computational metrics are based on examining how an algorithm interactswith the computer hardware on which it runs. The significant metricsconsidered are instruction mix, caching properties and pattern groupsize, and the search text length. The instruction mix refers to the typeof hardware instructions required by the algorithm. A sophisticatedinstruction mix is indicative of algorithms that require special purposemachine instructions or hardware to work efficiently. An embodiment ofthe present invention has no special instruction mix requirements andcan run well on most general-purpose computers.

The caching properties of an algorithm can significantly affectperformance. The strongest indicators of caching performance are cachesize and data locality. Data locality is defined by the relative amountof sequential versus random memory access the algorithm performs in andout of the cache. An embodiment of the present invention jumps fromstate to state in the state table. These jumps are data driven, and areessentially random. If only a small part of the state table can fit inthe computer's cache, there are likely to be many cache misses, and acache miss may cost 10 times that of a cache hit. Therefore, theperformance of an embodiment of the present invention is very sensitiveto the size of the cache, and whether the state table can fit in thecache. Ideally, the cache should be large enough to hold the entirestate table and have room left to bring the search data into the cacheas needed. This ideal configuration would typically only happen forsmall state tables, or on systems with very large caches. The statetable size is proportional to the total number of characters in all ofthe patterns included in the search.

The last issue considered in measuring pattern matching performance isthe problem domain of the search. A pattern matcher is applied to definethe size of the search text, the number of patterns, and the frequencyof searches. The problem domain of a typical IDS requires searchingnetwork traffic for known attack patterns. The pattern group sizes usedin a typical IDS are up to one-thousand or more patterns. A typical IDSsearches network packets that average six-hundred to eight-hundred bytesin each search, and a typical IDS does this up to 200,000 times asecond. In between pattern searches, other operations occur in a typicalIDS that potentially flush the cache. Each time a pattern search isstarted, the cache has to be reloaded which can represent a noticeablesetup cost for each search. In direct contrast to this type of search isthe sequential search of a very large data stream. This type of searchallows the higher performance cached data accesses to greatly acceleratethe overall search performance. A dictionary test demonstrates thisbehavior.

Test Results

Snort is an exemplary IDS. Test results for the standard Aho-Corasickalgorithm previously used in Snort, an embodiment of the presentinvention using full matrix storage implemented in Snort, and anembodiment of the present invention using banded-row storage implementedin Snort are presented.

The testing was divided into two types of pattern search tests.Dictionary tests were used to demonstrate the relative merits of patternsearch engines. The dictionary test selected was used to provide a longstream of text to search, providing the software an opportunity toachieve the best possible caching performance. A network traffic capturefile was also used and demonstrated the relative performance of allthree versions of the Aho-Corasick algorithm as implemented in Snort.

The dictionary test selected 1000 patterns from throughout an onlineKing-James bible. These patterns were used in groups of ten toone-thousand to search through the 1903 Webster's unabridged dictionary,about 2.3 megabytes of data. This ensured that there were plenty ofsuccessful pattern matches, causing the test to provide completecoverage of the search routine by executing both byte testing andmatching pattern code.

The three versions of the Aho-Corasick algorithm tested included: (1)the standard version already in use in Snort, which treats the statetable as a full matrix, (2) an embodiment of the present invention usingfull matrix storage, and (3) an embodiment of the present inventionusing banded-row storage.

A test bed was developed that included several different compiledversions of each of the Aho-Corasick routines to be tested. Thecompilers used included Microsoft™ Visual C++™6.0 (VC60), Intel™ C 6.0,and Cygwin gcc 3.3.1. Using three different compilers provided broaderinsight into the performance and caching behavior of the searchalgorithm and storage methods.

All results are for the DFA version of the Aho-Corasick algorithm. Theexact results for the dictionary tests on all three compilers are shownin FIGS. 4, 5, and 6. Results normalized against the standardAho-Corasick algorithm are shown in FIGS. 7, 8, and 9. The legends forthe bar graphs in FIGS. 4-10 are the same. As shown in FIG. 4, AC 410refers to the results from standard version of the Aho-Corasickalgorithm already in use in Snort, AC OPT 420 refers to the results froman embodiment of the present invention using full matrix storage, and ACBAND 430 refers to the results from an embodiment of the presentinvention using banded-row storage. These tests were conducted usingsixteen bit state values on a Dell™ 8100 1.7 GHz system with one G-byteof RAM.

The optimized versions performed significantly better than the standardversion in the dictionary tests with all three compilers. Comparing thecompiler results in FIGS. 4, 5, and 6 shows that the Intel™ compilerranked best in raw performance. The gcc compiler performed better thanthe VC60 compiler with the standard Aho-Corasick, but the VC60 compilerperformed better than the gcc compiler with an embodiment of the presentinvention using full matrix storage and an embodiment of the presentinvention using banded-row storage.

All three compilers produced similar performance trends. There is atleast a 1.4 times speed improvement of an embodiment of the presentinvention using full matrix storage over the standard Aho-Corasickalgorithm, except for the case of the VC60 compiler, where there is asmuch as a 2.5 times improvement on smaller pattern groups. An embodimentof the present invention using banded-row storage also produces asignificant speedup. In fact, an embodiment of the present inventionusing banded-row storage performed the best on the largest patterngroup. All three compilers showed an embodiment of the present inventionusing full matrix storage to be faster than an embodiment of the presentinvention using banded-row storage for all pattern group sizes up tofive-hundred patterns. At one-thousand patterns, an embodiment of thepresent invention using banded-row storage outperformed an embodiment ofthe present invention using full matrix storage for all three compilers.

The storage requirements for the different pattern group sizes are shownin FIG. 10, with the tabular results shown in FIG. 11. These resultsreflect a sixteen bit state size. An embodiment of the present inventionusing full matrix storage uses one-half the memory of the standardAho-Corasick algorithm, due to its use of a sixteen bit word size forstate values. An embodiment of the present invention using banded-rowstorage uses about one-fifteenth the storage of the standardAho-Corasick algorithm, and about one-seventh the storage of anembodiment of the present invention using full matrix storage.

An embodiment of the present invention using full matrix storage and anembodiment of the present invention using banded-row storage can useeither sixteen bit state or thirty-two bit state values. The sixteen bitstate values are limited to 2¹⁶ or sixty-five thousand states. Thethirty-two bit state values are limited to 2³² or 4 billion states. Themerit of using sixteen bit states is reduced memory consumption. FIGS.12, 13, and 14 compare the performance of the sixteen bit (AC OPT) andthe thirty-two bit (AC OPT 32) versions of an embodiment of the presentinvention using full matrix storage for the three compilers. The legendsfor the bar graphs in FIGS. 12-14 are the same. As shown in FIG. 12, ACOPT 1210 refers to the results from the sixteen bit version of anembodiment of the present invention using full matrix storage and AC OPT32 1220 refers to the thirty-two bit version of an embodiment of thepresent invention using full matrix storage. FIGS. 15, 16, and 17compare the performance of the sixteen bit (AC BANDED) and thethirty-two bit (AC BANDED 32) versions of an embodiment of the presentinvention using banded-row storage for the three compilers. The legendsfor the bar graphs in FIGS. 15-17 are the same. As shown in FIG. 15, ACBANDED 1510 refers to the results from the sixteen bit version of anembodiment of the present invention using banded-row storage and ACBANDED 32 1520 refers to the thirty-two bit version of an embodiment ofthe present invention using banded-row storage.

There are only minor differences between the sixteen and thirty-two bitversions of both an embodiment of the present invention using fullmatrix storage and an embodiment of the present invention usingbanded-row storage. The VC60 compiler demonstrates the largestdifference across all pattern sizes and favors the 32 bit state size.

Snort processes packets on standard Ethernet based networks that are upto 1460 bytes of data per packet. This type of data tests searchperformance and search setup costs on smaller search texts, and in arelatively cache unfriendly environment. The tests were performed byreplaying previously captured traffic directly into Snort. The compilerused was the Intel™ C 6.0 compiler on Linux. The test file selectedrepresented about two G-bytes of web centric traffic. This test file wasselected since Snort allows the selection of the amount of trafficinspected in each web request. The test worked as follows. Snort wasconfigured to inspect the first three-hundred bytes of each web pagerequest. This is typically how Snort might be configured. The Unix timecommand was used and the file was processed as usual, noting the userprocessing time. Snort then was configured to inspect each of the webpages completely as requested from the server. Once again, Snort was runand the user processing time was noted. The difference in timesrepresented the pure pattern matching time required to pattern match theextra data. This test did not measure absolute performance. It did showthe relative performance differences of each search engine in processingthe additional data.

The test file test was run with each version of the Aho-Corasickalgorithm and the Wu-Manber algorithm. The Wu-Manber algorithm in Snortis generally a very fast algorithm on average, but does not have a goodworst-case scenario. It is included to show the relative merits of itsgood average case performance compared to the Aho-Corasick algorithms.

The time differences, computed speed, and performance increase over thestandard Aho-Corasick is shown in FIG. 18 and FIG. 19. In FIGS. 18-19,WU-MAN refers to the results from the Wu-Manber algorithm, AC refers tothe results from the standard version of the Aho-Corasick algorithmalready in use in Snort, AC OPT refers to the results from an embodimentof the present invention using full matrix storage, and AC BAND refersto the results from an embodiment of the present invention usingbanded-row storage. These tests used a sixteen bit state size, were runon a system with dual 2.4 GHz Xeon™ cpus, and two G-bytes of RAM.

An embodiment of the present invention using full matrix storage and anembodiment of the present invention using banded-row storage aresignificantly faster than the standard Aho-Corasick algorithm and theWu-Manber algorithm. The Web rules in Snort contain many small two bytepatterns. This prevents the Wu-Manber algorithm from benefiting from abad character shift strategy.

An embodiment of the present invention using full matrix storage isthirty-one percent ( 8.5/12.4) faster, which means the engine canperform forty-six percent ( 12.4/8.5) more pattern matching in the sametime, and requires one-half the memory of original algorithm. Anembodiment of the present invention using banded-row storage isseventeen percent faster and yields a twenty percent performance gain,and uses about one-fourth of the memory of the standard Aho-Corasickalgorithm.

In accordance with an embodiment of the present invention, instructionsconfigured to be executed by a processor to perform a method are storedon a computer-readable medium. The computer-readable medium can be adevice that stores digital information. For example, a computer-readablemedium includes a compact disc read-only memory (CD-ROM) as is known inthe art for storing software. The computer-readable medium is accessedby a processor suitable for executing instructions configured to beexecuted. The terms “instructions configured to be executed” and“instructions to be executed” are meant to encompass any instructionsthat are ready to be executed in their present form (e.g., machine code)by a processor, or require further manipulation (e.g., compilation,decryption, or provided with an access code, etc.) to be ready to beexecuted by a processor.

Methods in accordance with an embodiment of the present inventiondisclosed herein can advantageously improve the performance of patternmatching in an IDS. One pattern matching method uses an optimized vectorimplementation of the Aho-Corasick state table that significantlyimproves performance. Another method uses sparse matrix storage toreduce memory requirements and further improve performance on largepattern groups.

The foregoing disclosure of the preferred embodiments of the presentinvention has been presented for purposes of illustration anddescription. It is not intended to be exhaustive or to limit theinvention to the precise forms disclosed. Many variations andmodifications of the embodiments described herein will be apparent toone of ordinary skill in the art in light of the above disclosure. Thescope of the invention is to be defined only by the claims appendedhereto, and by their equivalents.

Further, in describing representative embodiments of the presentinvention, the specification may have presented the method and/orprocess of the present invention as a particular sequence of steps.However, to the extent that the method or process does not rely on theparticular order of steps set forth herein, the method or process shouldnot be limited to the particular sequence of steps described. As one ofordinary skill in the art would appreciate, other sequences of steps maybe possible. Therefore, the particular order of the steps set forth inthe specification should not be construed as limitations on the claims.In addition, the claims directed to the method and/or process of thepresent invention should not be limited to the performance of theirsteps in the order written, and one skilled in the art can readilyappreciate that the sequences may be varied and still remain within thespirit and scope of the present invention.

1-17. (canceled)
 18. A method for searching for search patterns in atext sequence using a state table of a state machine algorithm in anintrusion detection system, wherein the state table includes a statetransition table and an array of per state pattern matching lists,comprising: reading a character from the text sequence; reading anelement of a current state vector of a current state of the statetransition table that corresponds to the character; if the element has anonzero value, checking a pattern matching element of the current statevector for a matching pattern flag; if the matching pattern flag is set,processing the patterns in the text sequence that match pattern matchinglists of the current state; selecting the current state vectorcorresponding to the nonzero value of the element; and reading a nextcharacter from the text sequence, wherein the state transition table andthe array of per state pattern matching lists are separate datastructures, the state transition table has a list of pointers to statevectors, each state vector representing valid transitions for the state.19. The method of claim 18, further comprising converting the characterto uppercase before the reading of the element of the state transitiontable and before the reading of the next character.
 20. The method ofclaim 18, wherein each state vector further includes an indicator thatidentifies a storage format of the element of the state transition tableas being in a full vector form where a state vector with a predeterminedfull number of elements or in a banded-row form where a state vector hasa variable number of elements reduced from the predetermined fullnumber, further comprising: checking the indicator of the current statevector that identifies the storage format; if the storage format isbanded-row form, determining if the character is in a band from anelement of the current state vector identifying a number of elements inthe band, an element of the current state vector identifying an index ofa first element of the band, and elements of the band; then, if thecharacter is in the band, selecting the current state vectorcorresponding to the nonzero value of the element; otherwise if thecharacter is not in the band, returning the current state vector to aninitial state; otherwise handling the character in accordance with afull vector format.
 21. The method of claim 19, further comprising: ifthe matching pattern flag is set, comparing a case sensitive version ofthe pattern match from the text sequence and a case sensitive version ofa search pattern; and if the matching pattern flag is set and if thecase sensitive version of the pattern match from the text sequence andthe case sensitive version of a search pattern match, processing thecase sensitive version of the pattern match.
 22. The method of claim 18,the state machine algorithm comprising Aho-Corasick.
 23. (canceled) 24.A state table for a state machine algorithm used in a pattern matchingapplication, comprising: a state transition table; an array of per statematching pattern lists; and a failure pointer list for each state for anon-deterministic finite automata.
 25. The state table of claim 24, thestate transition table comprising a state vector for each state.
 26. Thestate table of claim 25, the state vector being stored in a sparsevector format.
 27. The state table of claim 25, the sparse vector formatcomprising one of compressed sparse vector format, sparse-row format,and banded-row format.
 28. The state table of claim 24, the statemachine algorithm comprising Aho-Corasick.
 29. The state table of claim24, the pattern matching application comprising one of an intrusiondetection system, a passive network monitor, an active network monitor,a protocol analyzer, and a search engine.
 30. The method of claim 18,wherein: the state transition table is stored in is stored in cachememory, the array of per state pattern matching lists is stored in amemory different from the cache memory, the reading of the element ofthe current state vector of the state transition table is done from thestate transition table in the cache memory, and the processing with thepattern matching lists is done from the per state pattern matching listsin the memory different from the cache memory.
 31. The method of claim18, wherein each of the elements in the state transition table has asize of two bytes.
 32. The method of claim 20, wherein the state vectorin full vector form holds two hundred fifty six transition elements. 33.A computer-readable medium comprising instructions being executed by acomputer, the instructions including a computer-implemented method forsearching for search patterns in a text sequence using a state table ofa state machine algorithm in an intrusion detection system, wherein thestate table includes a state transition table and an array of per statepattern matching lists, the instructions for implementing the steps of:reading a character from the text sequence; reading an element of acurrent state vector of a current state of the state transition tablethat corresponds to the character; if the element has a nonzero value,checking a pattern matching element of the current state vector for amatching pattern flag; if the matching pattern flag is set, processingthe patterns in the text sequence that match pattern matching lists ofthe current state; selecting the current state vector corresponding tothe nonzero value of the element; and reading a next character from thetext sequence, wherein the state transition table and the array of perstate pattern matching lists are separate data structures, the statetransition table has a list of pointers to state vectors, each statevector representing valid transitions for the state.
 34. Thecomputer-readable medium of claim 33, further comprising converting thecharacter to uppercase before the reading of the element of the statetransition table and before the reading of the next character.
 35. Thecomputer-readable medium of claim 34, further comprising: if thematching pattern flag is set, comparing a case sensitive version of thepattern match from the text sequence and a case sensitive version of asearch pattern; and if the matching pattern flag is set and if the casesensitive version of the pattern match from the text sequence and thecase sensitive version of a search pattern match, processing the casesensitive version of the pattern match.
 36. The computer-readable mediumof claim 33, wherein each state vector further includes an indicatorthat identifies a storage format of the element of the state transitiontable as being in a full vector form where a state vector with apredetermined full number of elements, or in a banded-row form where astate vector has a variable number of elements reduced from thepredetermined full number, further comprising: checking the indicator ofthe current state vector that identifies the storage format; if thestorage format is banded-row form, determining if the character is in aband from an element of the current state vector identifying a number ofelements in the band, an element of the current state vector identifyingan index of a first element of the band, and elements of the band; then,if the character is in the band, selecting the current state vectorcorresponding to the nonzero value of the element; otherwise if thecharacter is not in the band, returning the current state vector to aninitial state; otherwise handling the character in accordance with afull vector format.
 37. The computer-readable medium of claim 33,wherein: the state transition table is stored in is stored in cachememory, the array of per state pattern matching lists is stored in amemory different from the cache memory, the reading of the element ofthe current state vector of the state transition table is done from thestate transition table in the cache memory, and the processing with thepattern matching lists is done from the per state pattern matching listsin the memory different from the cache memory.